test(rocm): ReshapeAndCache->PagedAttention composition at real dims (issue #41) - #497
Draft
VikashLoomba wants to merge 2 commits into
Draft
test(rocm): ReshapeAndCache->PagedAttention composition at real dims (issue #41)#497VikashLoomba wants to merge 2 commits into
VikashLoomba wants to merge 2 commits into
Conversation
localai-org-maint-bot
suggested changes
Aug 14, 2026
localai-org-maint-bot
left a comment
localai-org-maint-bot
left a comment
Contributor
There was a problem hiding this comment.
The new composition test does not build a self-consistent logical sequence. ReshapeAndCache writes tokens to (i*7+3)%64, while PagedAttention reads logical positions 0..19 through block table {3,1,2,0} (physical slots 48..63 and 16..19). Most read positions were never written, so CPU/device agreement can compare zeros and miss the intended scatter/layout defect. Please derive each token slot from its logical position and the shuffled block table (for example block_table[i/BS]*BS + i%BS), then add a mutation/guard proving a wrong physical-block mapping makes the test fail. Also remove the unused padded qstride: the tensor is declared contiguous, so those extra uploaded elements do not exercise a fused-view stride.
…udler#41) The in-tree paged-attention case hand-builds a contiguous KV cache; the real model path writes it with ReshapeAndCache and reads it back. This case is that composition at real model dims (Dh=256, Hq=8, Hkv=2, block_size 16) with a shuffled block table and non-sequential slot mapping — the layout a stride/scatter bug would live in and the contiguous case cannot see. Surfaced by the mudler#41 Qwen3.5-0.8B divergence investigation: every compositional piece of the ROCm attention path now validates in isolation, which is what localizes the residual divergence to bf16-softmax accumulation rather than a kernel defect. Evidence (4x gfx1100, ROCm 7.14, Release): the new case passes 7/7 vs the CPU oracle; full cross-device suite green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
… table + an anti-vacuity guard -- the mudler#497 review rework CHANGES_REQUESTED review (localai-org-maint-bot, 2026-08-14), both findings accepted: 1. The first version scattered token i to slot (i*7+3)%64 while the attention read walked logical positions through block table {3,1,2,0} — the write set and the read set were disjoint by construction, so most positions compared unwritten zeros on both backends and a scatter/layout defect could not have failed the case. The slot mapping now derives from the logical position through the shuffled table (slots[i] = block_table[i/BS]*BS + i%BS), which is the engine's real mapping. 2. Anti-vacuity guard: a corrupted block table (the first two LOGICAL blocks swapped — both hold real tokens) must change the attention output; if the composition compared only unwritten slots it would not. (The first draft of the guard swapped two blocks OUTSIDE the logical range and was itself vacuous — the guard proved the guard; the committed version swaps the in-range mapping.) Also removed the unused padded qstride — the query tensor is declared contiguous, so the padding exercised nothing. Gates (gfx1100, flock): test_backend_cross_device 20/20 with the corrected composition. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
VikashLoomba
force-pushed
the
row/ROCM-ATTN-COMPOSE-TEST
branch
from
August 14, 2026 09:07
6104ca1 to
8edebbd
Compare
Contributor
Author
|
Both findings accepted and reworked (commit 8edebbd, rebased onto current main):
Gate: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row
BACKEND-ROCM— test-hardening only (one additive cross-device case). Issue #41.What changed
Adds a cross-device case for the KV-cache composition the in-tree suite doesn't cover: the existing paged-attention case hand-builds a contiguous cache, but the real model path writes KV via
ReshapeAndCacheand reads it back viaPagedAttention. This case is that composition at real Qwen3.5-0.8B dims (Dh=256, Hq=8, Hkv=2, block_size 16), a shuffled block table, and a non-sequential slot mapping — the layout a stride/scatter bug would live in and the contiguous case cannot see.Surfaced by the #41 Qwen3.5-0.8B divergence investigation: with this composition passing, every piece of the ROCm attention path validates in isolation, which is what localizes the residual divergence to bf16-softmax accumulation rather than a kernel defect (full causal chain in this #41 comment).
Evidence (4× gfx1100, ROCm 7.14, Release)
agent-preflight.sh --stagedgreen;check-commit-trailersgreenSpeed claims
Honest gaps